Continuing Lane Driving Prediction

ABSTRACT

The technology relates to controlling a vehicle in an autonomous driving mode in accordance with behavior predictions for other road users in the vehicle&#39;s vicinity. In particular, the vehicle&#39;s onboard computing system may predict whether another road user will perform a “continuing” lane driving operation, such as going straight in a turn-only lane. Sensor data from detected/observed objects in the vehicle&#39;s nearby environment may be evaluated in view of one or more possible behaviors for different types of objects. In addition, roadway features, in particular whether lane segments are connected in a roadgraph, are also evaluated to determine probabilities of whether other road users may make an improper continuing lane driving operation. This is used to generate more accurate behavior predictions, which the vehicle can use to take alternative (e.g., corrective) driving actions.

BACKGROUND

Vehicles that operate in an autonomous driving mode may transportpassengers or items such as cargo from one location to another. Suchvehicles are typically equipped with various types of sensors in orderto detect objects in the surroundings. For example, self-drivingvehicles may include lasers (lidars), radar, cameras, and other deviceswhich obtain sensor data (e.g., point cloud data or images) from thevehicle's surroundings. Sensor data from one or more of these devicesmay be used to detect objects and their respective characteristics(e.g., position, shape, heading, speed, etc.). These characteristics canbe used to predict what an object is likely to do for some brief periodof time in the immediate future.

The predictions can aid in controlling the vehicle in order to avoidthese objects. Thus, detection, identification, and prediction arecritical functions for the safe operation of autonomous vehicle.However, it can be very challenging to predict when another road user orother agent such as a pedestrian may not follow the rules of the road.Therefore, it is difficult for the autonomous vehicle to plan foralternative driving operations.

BRIEF SUMMARY

The technology relates to controlling a vehicle in an autonomous drivingmode in accordance with behavior predictions for other road users in thevehicle's vicinity. For instance, such autonomous vehicles operate bydetecting and identifying objects in the vehicle's nearby environmentand reacting to those objects, for instance by changing lanes, speedingup or slowing down. In order to control the vehicle safely, thevehicle's computing devices should be able to predict what other roadusers are likely to do in the near future, such as over the next 5-10seconds (or more or less). This can include determining the likelihoodsthat another object will take different actions, even if those actionsare atypical or violate the rules of the road. For instance, apedestrian may jaywalk or a car may run a red light.

One particular situation of interest involves “continuing” lane driving.For instance, a motorcycle may be in a left turn only lane, but ends updriving straight along its prior trajectory instead of making the turn.Predicting a likelihood of and the geometry for continuing lane drivingcan be very beneficial for autonomous vehicle operation, includingavoiding rapid changes in acceleration (“jerk”) due to braking to avoidthe other vehicle.

According to one aspect, a method of operating a vehicle in anautonomous driving mode is provided. The method comprises identifying,by one or more processors, a source lane along a roadway, the sourcelane being associated with a road user other than the vehicle;identifying, by the one or more processors according to a first trainedmachine learning model, a target continuing lane, the target continuinglane being an impermissible lane that the road user may move toward;receiving, by one or more sensors of a perception system of the vehicle,sensor data associated with objects in an external environment of thevehicle including the road user; generating, by the one or moreprocessors according to a second trained machine learning model and thereceived sensor data, a continuing lane driving behavior prediction forthe road user along the continuing lane; and controlling, by the onemore processors, the vehicle in the autonomous driving mode based on thecontinuing lane driving behavior prediction.

In one example, identifying the source lane includes fetching mapinformation within a curtain radius of a current location of thevehicle. Alternatively or additionally, identifying the targetcontinuing lane includes identifying a set of possible driving locationsexiting from the source lane.

In another example, prior to controlling the vehicle, the method furtherincludes generating a motion plan for the vehicle based at least in parton the continuing lane driving behavior prediction. The method mayfurther comprise, prior to controlling the vehicle, assigning alikelihood to the continuing lane driving behavior prediction. Thisapproach may also include generating one or more continuing lane drivingbehavior predictions based on the assigned likelihood of the continuinglane driving behavior prediction.

The target continuing lane may either (i) continue straight fromdesignated a turn-only lane, or (ii) make a turn from a lane that isdesignated to continue straight only.

In an example, identifying the target continuing lane includes:identifying any candidate continuing lanes; evaluating all candidatecontinuing lanes by the first trained machine learning model to identifya score for each candidate continuing lane; and evaluating whether thescore for a given one of the candidate continuing lanes exceeds athreshold and is higher than the scores for all other candidatecontinuing lanes. Here, when the score for the given candidatecontinuing lane exceeds the threshold and is higher than the scores forall other candidate continuing lanes, the method may include identifyingthe given candidate continuing lane as the target continuing lane.Alternatively or additionally when the score for the given candidatecontinuing lane does not exceed the threshold or is not higher than thescores for all other candidate continuing lanes, the method may notidentify the given candidate continuing lane as the target continuinglane.

In a further example, the first trained machine learning model istrained according to one or more source lane features and one or moretarget lane features. Here, the one or more source lane features and theone or more target lane features may each include at least one of a laneheading, a speed limit, lane curvature, lane incline, lane camber,signage, a traffic light, or an exit point. The first trained machinelearning model may be further trained according to one or additionalfeatures associated with both a source lane and a target lane. The oneor more additional features may include at least one of a headingdifference between the source lane and the target lane, a distancebetween the source lane and the target lane, or a lateral distancebetween the source lane and the target lane.

In yet another example, generating the continuing lane driving behaviorprediction includes: running the second trained machine learning modelfor each candidate continuing lane to identify all candidate continuinglane actions for the road user; evaluating all candidate continuing laneactions to identify a score for each candidate continuing lane action;and evaluating whether the score for a given one of the candidatecontinuing lane actions exceeds a threshold and is higher than thescores for all other candidate continuing lane actions.

When the score for the given candidate continuing lane action exceedsthe threshold and is higher than the scores for all other candidatecontinuing lane actions, the method may include generating thecontinuing lane driving behavior prediction according to the givencandidate continuing lane action. Alternatively or additionally, whenthe score for the given candidate continuing lane action does not exceedthe threshold or is not higher than the scores for all other candidatecontinuing lane actions, the method may not generate the continuing lanedriving behavior prediction according to the given candidate continuinglane action

In another example, the second trained machine learning model is trainedaccording to one or more lane features and one or more agent features.Here, the one or more agent features are associated with the road user.Here, the one or more lane features may include at least one of a laneheading, a speed limit, lane curvature, lane incline, lane camber,signage, a traffic light, or an exit point; and the one or more agentfeatures may include at least one of an agent history, a currentlocation, speed, acceleration, heading, use of a turn signal or use of ahazard signal. The second trained machine learning model may be furthertrained according to one or additional features associated with both thelane features and the agent features. And the one or more additionalfeatures may include at least one of a distance between the currentlocation of the road user and the lane, or a difference between theheading of the road user and the heading of the lane.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-B illustrate example autonomous passenger vehicles inaccordance with aspects of the technology.

FIGS. 1C-D illustrate an example autonomous cargo-type vehicleconfigured for use with aspects of the technology.

FIG. 2 illustrates components of an autonomous vehicle in accordancewith aspects of the technology.

FIGS. 3A-B are block diagrams of systems of an example autonomouscargo-type vehicle in accordance with aspects of the technology.

FIGS. 4A-B illustrate general turning scenarios in accordance withaspects of the technology.

FIGS. 5A-C illustrate continuing lane driving scenarios in accordancewith aspects of the technology.

FIG. 6 illustrates an example process in accordance with aspects of thetechnology.

FIGS. 7A-B illustrate aspects of the process of FIG. 6 in accordancewith aspects of the technology.

FIGS. 8A-B illustrate features that can be stored and used as traininginputs in accordance with aspects of the technology.

FIGS. 9A-B illustrate an example system in accordance with aspects ofthe technology.

FIG. 10 illustrates an example process in accordance with aspects of thetechnology.

DETAILED DESCRIPTION Overview

Identifying unpermitted continuing lane driving likelihoods is achallenging problem in behavior prediction, as it not only involves theprediction of whether the other road user would go from a turn only laneto a driving straight lane, but also involves identifying the targetlane(s) the other road user can go to. Improper continuing lane driving(that is not consistent with the rules of the road) can involve a roaduser such as a cyclist or a vehicle (e.g., a motorcyclist, car, truck,bus, emergency vehicle etc.) that continues straight from a turn-onlylane, and a map (e.g., a roadgraph) does not connect the turn only lane(e.g., the source lane) with the lane the other road user goes to (e.g.,the target lane). The destination lane is called the continuing lane andthe behavior is called continuing lane driving. Since the roadgraph doesnot connect the turn only (source) lane with the continuing (target)lane, this type of behavior is considered improper to perform.

One approach towards addressing this problem can involve the use ofheuristics, e.g., the other object(s) heading and speed, roadproperties, etc. This approach could work in many cases but has certainlimitations, including that heuristics are brittle and are often basedon a small dataset, there is low precision/recall, the geolocation isoverfitted, and delays in predictions. Examples of road properties caninclude lane heading, lane speed limits, traffic signals, stop or yieldsigns, the lane's entry and exit points, etc. Brittleness here meansthat the heuristics are overfitted to specific use cases and are notable to handle cases with even slight variations from them.Precision/recall here refers to the precision and recall of theheuristics to predict that the cyclist, vehicle or other road user isgoing to follow the continuing lane. Overfitting to geolocations meansthat the heuristics would be overfitted to regions in which autonomousvehicles drive and see bugs or other spurious input.

In contrast to a heuristic approach, machine learning models are moregeneralized and can be used for any location/region. According toaspects of the technology, a machine learning approach is employed inorder to improve the predictions and generate better behaviorpredictions for continuing lane driving situations. The system employsone or more machine learning models that determine whether a given laneis a continuing lane for a turn, or a shared turn lane, in other wordsdetermine whether a lane is the target lane for a turn only lane or ashared turn lane. The machine learning model(s) also determine whetherthe other road user is moving, or intends to move, to the continuinglane from turn-only lane. This approach is beneficial for severalreasons including (i) high precision/recall (higher accuracy), (ii) itprovides a generalized solution application for various geolocations,(iii) it gives early predictions even before reaching the intersection(e.g., by evaluating earlier behaviors before getting to theintersection; considering the actions of similar road users at theintersection; analyzing the traffic light state; and/or looking at roaduser behavior at different geographic locations), and (iv) the ease ofmaintainability and scaling, with the ability to evaluate a massiveamounts of data (e.g., years' worth of data).

Example Vehicle Systems

The technology may be employed in all manner of self-driving vehicles,including vehicles that transport passengers or items such as fooddeliveries, packages, cargo, etc. While certain aspects of thedisclosure may be particularly useful in connection with specific typesof vehicles, the vehicle may be different types of vehicle including,but not limited to, cars, vans, motorcycles, cargo vehicles, buses,recreational vehicles, emergency vehicles, construction equipment, etc.

FIG. 1A illustrates a perspective view of an example passenger vehicle100, such as a minivan or sport utility vehicle (SUV). FIG. 1Billustrates a perspective view of another example passenger vehicle 140,such as a sedan. The passenger vehicles may include various sensors forobtaining information about the vehicle's external environment. Forinstance, a roof-top housing unit (roof pod assembly) 102 may include alidar sensor as well as various cameras (e.g., optical or infrared),radar units, acoustical sensors (e.g., microphone or sonar-typesensors), inertial (e.g., accelerometer, gyroscope, etc.) or othersensors (e.g., positioning sensors such as GPS sensors). Housing 104,located at the front end of vehicle 100, and housings 106 a, 106 b onthe driver's and passenger's sides of the vehicle may each incorporatelidar, radar, camera and/or other sensors. For example, housing 106 amay be located in front of the driver's side door along a quarter panelof the vehicle. As shown, the passenger vehicle 100 also includeshousings 108 a, 108 b for radar units, lidar and/or cameras also locatedtowards the rear roof portion of the vehicle. Additional lidar, radarunits and/or cameras (not shown) may be located at other places alongthe vehicle 100. For instance, arrow 110 indicates that a sensor unit(not shown) may be positioned along the rear of the vehicle 100, such ason or adjacent to the bumper. Depending on the vehicle type and sensorhousing configuration(s), acoustical sensors may be disposed in any orall of these housings around the vehicle.

Arrow 114 indicates that the roof pod 102 as shown includes a basesection coupled to the roof of the vehicle. And arrow 116 indicated thatthe roof pod 102 also includes an upper section raised above the basesection. Each of the base section and upper section may house differentsensor units configured to obtain information about objects andconditions in the environment around the vehicle. The roof pod 102 andother sensor housings may also be disposed along vehicle 140 of FIG. 1B.By way of example, each sensor unit may include one or more sensors ofthe types described above, such as lidar, radar, camera (e.g., opticalor infrared), acoustical (e.g., a passive microphone or active soundemitting sonar-type sensor), inertial (e.g., accelerometer, gyroscope,etc.) or other sensors (e.g., positioning sensors such as GPS sensors).

FIGS. 1C-D illustrate an example cargo vehicle 150, such as atractor-trailer truck. The truck may include, e.g., a single, double ortriple trailer, or may be another medium or heavy duty truck such as incommercial weight classes 4 through 8. As shown, the truck includes atractor unit 152 and a single cargo unit or trailer 154. The trailer 154may be fully enclosed, open such as a flat bed, or partially opendepending on the type of cargo to be transported. In this example, thetractor unit 152 includes the engine and steering systems (not shown)and a cab 156 for a driver and any passengers.

The trailer 154 includes a hitching point, known as a kingpin, 158. Thekingpin 158 is typically formed as a solid steel shaft, which isconfigured to pivotally attach to the tractor unit 152. In particular,the kingpin 158 attaches to a trailer coupling 160, known as afifth-wheel, that is mounted rearward of the cab. For a double or tripletractor-trailer, the second and/or third trailers may have simple hitchconnections to the leading trailer. Or, alternatively, each trailer mayhave its own kingpin. In this case, at least the first and secondtrailers could include a fifth-wheel type structure arranged to coupleto the next trailer.

As shown, the tractor may have one or more sensor units 162, 164disposed therealong. For instance, one or more sensor units 162 may bedisposed on a roof or top portion of the cab 156, and one or more sidesensor units 164 may be disposed on left and/or right sides of the cab156. Sensor units may also be located along other regions of the cab156, such as along the front bumper or hood area, in the rear of thecab, adjacent to the fifth-wheel, underneath the chassis, etc. Thetrailer 154 may also have one or more sensor units 166 disposedtherealong, for instance along a side panel, front, rear, roof and/orundercarriage of the trailer 154.

As with the sensor units of the passenger vehicles of FIGS. 1A-B, eachsensor unit of the cargo vehicle may include one or more sensors, suchas lidar, radar, camera (e.g., optical or infrared), acoustical (e.g.,microphone or sonar-type sensor), inertial (e.g., accelerometer,gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPSsensors).

There are different degrees of autonomy that may occur for aself-driving vehicle operating in a partially or fully autonomousdriving mode. The U.S. National Highway Traffic Safety Administrationand the Society of Automotive Engineers have identified different levelsto indicate how much, or how little, the vehicle controls the driving.For instance, Level 0 has no automation and the driver makes alldriving-related decisions. The lowest semi-autonomous mode, Level 1,includes some drive assistance such as cruise control. At this level,the vehicle may operate in a strictly driver-information system withoutneeding any automated control over the vehicle. Here, the vehicle'sonboard sensors, relative positional knowledge between them, and a wayfor them to exchange data, can be employed to implement aspects of thetechnology as discussed herein. Level 2 has partial automation ofcertain driving operations, while Level 3 involves conditionalautomation that can enable a person in the driver's seat to take controlas warranted. In contrast, Level 4 is a high automation level where thevehicle is able to drive without assistance in select conditions. AndLevel 5 is a fully autonomous mode in which the vehicle is able to drivewithout assistance in all situations. The architectures, components,systems and methods described herein can function in any of the semi orfully-autonomous modes for which speed planning is employed, which arereferred to herein as autonomous driving modes. Thus, reference to anautonomous driving mode herein includes both partial and full autonomy.

FIG. 2 illustrates a block diagram 200 with various components andsystems of an exemplary vehicle, such as passenger vehicle 100 or 140,to operate in an autonomous driving mode. As shown, the block diagram200 includes one or more computing devices 202, such as computingdevices containing one or more processors 204, memory 206 and othercomponents typically present in general purpose computing devices. Thememory 206 stores information accessible by the one or more processors204, including instructions 208 and data 210 that may be executed orotherwise used by the processor(s) 204. The computing system may controloverall operation of the vehicle when operating in an autonomous drivingmode.

The memory 206 may be of any type capable of storing informationaccessible by the processor, including a computing device-readablemedium. The memory is a non-transitory medium such as a hard-drive,memory card, optical disk, solid-state, etc. Systems may includedifferent combinations of the foregoing, whereby different portions ofthe instructions and data are stored on different types of media. Theinstructions 208 may be any set of instructions to be executed directly(such as machine code) or indirectly (such as scripts) by theprocessor(s). For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions”, “modules” and “programs” may be usedinterchangeably herein. The instructions may be stored in object codeformat for direct processing by the processor, or in any other computingdevice language including scripts or collections of independent sourcecode modules that are interpreted on demand or compiled in advance. Inone example, some or all of the memory 206 may be an event data recorderor other secure data storage system configured to store vehiclediagnostics and/or detected sensor data, which may be on board thevehicle or remote, depending on the implementation.

The data 210 can include, e.g., map information (e.g., roadgraphs), roaduser behavior models and/or behavior-time models, dynamic or staticvehicle models, etc., which may be retrieved, stored or modified by oneor more processors 204 in accordance with the instructions 208. In oneexample, behavior-related models and/or continuing lane models may beused to predict the future behavior of one or more detected objects inthe vehicle's nearby environment for a pre-determined period of time,such as the next 10 seconds or more or less.

In one example, the models may be configured to use data for an objectreceived from the perception system, and in particular another roaduser, including the road user's characteristics as well as additionalcontextual information. As an example, given the current location,heading, speed, and other characteristics included in the data from thevehicle's onboard perception system (e.g., historical information fromthe last 5-30 seconds), a given model may provide a set of one or morepredictions for how the object could behave for the predetermined periodof time as well as a corresponding likelihood value for each prediction.The predictions may include a trajectory, for instance, defining a setof future locations where the object is expected to be at various timesin the future corresponding to the predetermined period of time. Thelikelihood values may indicate which of the predictions are more likelyto occur (relative to one another). In this regard, the prediction withthe greatest likelihood value may be the most likely to occur whereaspredictions with lower likelihood values may be less likely to occur.This approach may be used by the processing system of the autonomousvehicle to make continuing lane-related decisions, for instance to avoida road user that is predicted to continue straight instead of turning ina turn-only lane.

Thus, the models may be configured to generate a set of possiblehypotheses for what a particular road user will do over a particularhorizon or predetermined period of time (e.g. 10 seconds) and relativelikelihoods for each hypothesis (e.g., a likelihood of making a turningas well as a likelihood of continuing forward instead of turning. Thesemodels may be trained using data about how an object observed at thatlocation behaved in the past, roadgraphs indicating permissible andimpermissible lane options, etc., and may also be specificallydesignated for particular types of objects, such as sedans, buses,pedestrians, motorcycles, bicyclists, emergency vehicles, constructionvehicles, etc. The computing devices can then reason about hypothesesthat interact with the vehicle's trajectory and are of a sufficientlikelihood to be worth considering. Inputs to the models may includelane-specific features, agent-specific features and/or other features(see, e.g., FIG. 8B). For instance, lane-specific features can includeone or more of lane heading, speed limit, curvature, incline, camber,whether there is a stop sign or other traffic sign present, whetherthere is a traffic light, entry and/or exit point(s), etc.Agent-specific features can include one or more of history (e.g., whatthe agent has done in the last 5-30 seconds, or more or less), speed,acceleration, heading, use of turn signal or hazard signal, etc. Otherfeatures may include one or more of the distance between the agent'scurrent position and the lane of interest, the difference between theagent heading and lane heading, etc. In one scenario, Gradient BoostedDecision Trees (GDBT) models may be employed; however, in otherscenarios other techniques such as heat maps, Deep Neural Networks(DNNs), Convolutional Neural Networks (CNNs), global nets, and/or VectorNets could be employed. The models may be trained offline, for instanceusing a back-end remote computing system (see FIGS. 9A-B) or trained bythe vehicle's onboard computing system.

The processors 204 may be any conventional processors, such ascommercially available CPUs. Alternatively, each processor may be adedicated device such as an ASIC or other hardware-based processor.Although FIG. 2 functionally illustrates the processors, memory, andother elements of computing devices 202 as being within the same block,such devices may actually include multiple processors, computingdevices, or memories that may or may not be stored within the samephysical housing. Similarly, the memory 206 may be a hard drive or otherstorage media located in a housing different from that of theprocessor(s) 204. Accordingly, references to a processor or computingdevice will be understood to include references to a collection ofprocessors or computing devices or memories that may or may not operatein parallel.

In one example, the computing devices 202 may form an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may be capable of communicating with various componentsof the vehicle. For example, the computing devices 202 may be incommunication with various systems of the vehicle, including a drivingsystem including a deceleration system 212 (for controlling braking ofthe vehicle), acceleration system 214 (for controlling acceleration ofthe vehicle), steering system 216 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 218 (forcontrolling turn signals), navigation system 220 (for navigating thevehicle to a location or around objects) and a positioning system 222(for determining the position of the vehicle, e.g., including thevehicle's pose, e.g., position and orientation along the roadway orpitch, yaw and roll of the vehicle chassis relative to a coordinatesystem). The autonomous driving computing system may employ a plannermodule 223, in accordance with the navigation system 220, thepositioning system 222 and/or other components of the system, e.g., fordetermining a route from a starting point to a destination, fordetermining a speed plan for an upcoming timeframe (e.g., the next 5-20seconds), or for making modifications to various driving aspects in viewof current or expected environmental conditions.

The computing devices 202 are also operatively coupled to a perceptionsystem 224 (for detecting objects in the vehicle's environment andinformation about the vehicle such as tracking increases or decreases inspeed and the direction of such changes), a power system 226 (forexample, a battery and/or gas or diesel powered engine) and atransmission system 230 in order to control the movement, speed, etc.,of the vehicle in accordance with the instructions 208 of memory 206 inan autonomous driving mode which does not require or need continuous orperiodic input from a passenger of the vehicle. Some or all of thewheels/tires 228 are coupled to the transmission system 230, and thecomputing devices 202 may be able to receive information about tirepressure, balance and other factors that may impact driving in anautonomous mode.

The computing devices 202 may control the direction and speed of thevehicle, e.g., via the planner module 223, by controlling variouscomponents. By way of example, computing devices 202 may navigate thevehicle to a destination location completely autonomously using datafrom map information and navigation system 220. Computing devices 202may use the positioning system 222 to determine the vehicle's locationand the perception system 224 to detect and respond to objects whenneeded to reach the location safely. In order to do so, computingdevices 202 may implement a speed plan that causes the vehicle toaccelerate (e.g., by increasing fuel or other energy provided to theengine by acceleration system 214), decelerate (e.g., by decreasing thefuel supplied to the engine, changing gears, and/or by applying brakesby deceleration system 212), change direction (e.g., by turning thefront or other wheels of vehicle 100 by steering system 216), and signalsuch changes (e.g., by lighting turn signals of signaling system 218).Thus, the acceleration system 214 and deceleration system 212 may be apart of a drivetrain or other type of transmission system 230 thatincludes various components between an engine of the vehicle and thewheels of the vehicle. Again, by controlling these systems, computingdevices 202 may also control the transmission system 230 of the vehiclein order to maneuver the vehicle autonomously.

Navigation system 220 may be used by computing devices 202 in order todetermine and follow a route to a location. In this regard, thenavigation system 220 and/or memory 206 may store map information, whichthe computing devices 202 can use to navigate or control the vehicle.The map information may include information identifying the shape,location, and other characteristics of lanes, traffic signal lights,crosswalks, sidewalks, stop or yield signs, stop lines or other roadmarkings, etc. Areas where a vehicle can drive may indicate the locationand direction in which a vehicle should generally travel at variouslocations in the map information.

While the map information may be image-based maps, the map informationneed not be entirely image based (for example, raster). For instance,the map information may include one or more roadgraphs, graph networksor road networks of information such as roads, lanes, intersections, andthe connections between these features which may be represented by roadsegments. Each feature in the map may also be stored as graph data andmay be associated with information such as a geographic location andwhether or not it is linked to other related features, for example, atraffic light, a stop or yield sign or road markings such as stop linesand crosswalks may be linked to a road and an intersection, etc. In someexamples, the associated data may include grid-based indices of a roadnetwork to allow for efficient lookup of certain road network features.

In this regard, the map information may include a plurality of graphnodes and edges representing road or lane segments that together make upthe road network of the map information. In this case, each edge may bedefined by a starting graph node having a specific geographic location(e.g., latitude, longitude, altitude, etc.), an ending graph node havinga specific geographic location (e.g., latitude, longitude, altitude,etc.), and a direction. This direction may refer to a direction thevehicle must be moving in in order to follow the edge (i.e., a directionof traffic flow). The graph nodes may be located at fixed or variabledistances. For instance, the spacing of the graph nodes may range from afew centimeters to a few meters and may correspond to the speed limit ofa road on which the graph node is located. In this regard, greaterspeeds may correspond to greater distances between graph nodes.

Thus, the maps may identify the shape, elevation/incline, curvature andcamber of roadways, as well as the presence of lane markers,intersections, stop lines, crosswalks, posted speed limits, trafficsignal lights (e.g., including turn signals), buildings, signs, realtime traffic information, vegetation, or other such objects andinformation. The lane markers may include features such as solid orbroken double or single lane lines, solid or broken lane lines,reflectors, etc. A given lane may be associated with left and/or rightlane lines or other lane markers that define the boundary of the lane.Thus, most lanes may be bounded by a left edge of one lane line and aright edge of another lane line.

The perception system 224 includes sensors 232 for detecting objectsexternal to the vehicle. The detected objects may be other vehicles,obstacles in the roadway, traffic signals, signs, road markings (e.g.,crosswalks and stop lines), objects adjacent to the roadway such assidewalks, trees or shrubbery, etc. The sensors may 232 may also detectcertain aspects of weather conditions, such as snow, rain or waterspray, or puddles, ice or other materials on the roadway.

By way of example only, the sensors of the perception system may includelight detection and ranging (lidar) sensors, radar units, cameras (e.g.,optical imaging devices, with or without a neutral-density filter (ND)filter), positioning sensors (e.g., gyroscopes, accelerometers and/orother inertial components), infrared sensors, and/or any other detectiondevices that record data which may be processed by computing devices202. The perception system 224 may also include one or more microphonesor other acoustical arrays, for instance arranged along the roof pod 102and/or other sensor assembly housings.

Such sensors of the perception system 224 may detect objects outside ofthe vehicle and their characteristics such as location, orientation(pose) relative to the roadway, size, shape, type (for instance,vehicle, pedestrian, bicyclist, etc.), heading, speed of movementrelative to the vehicle, etc., as well as environmental conditionsaround the vehicle. The perception system 224 may also include othersensors within the vehicle to detect objects and conditions associatedwith the vehicle. For instance, sensors may detect, e.g., one or morepersons, pets or packages, within the vehicle. Other sensors may detectconditions within and/or outside the vehicle such as temperature,humidity, etc. Still other sensors 232 of the perception system 224 maymeasure the rate of rotation and angle of the wheels 228, an amount or atype of braking by the deceleration system 212, overall vehicle pose,and other factors associated with the equipment of the vehicle itself.

The raw data and/or characteristics of objects received from theperception system may be used with contextual information as input to abehavior model of data 210 to make a prediction about what other objectsare going to do for the predetermined period of time. For instance,information such as the object's type, location, pose along the roadway,recent motion heading, acceleration, and velocity may be combined withother information such as where the object is in the world using thedetailed map information discussed above may be used as input to abehavior model. The contextual information may include the status ofother objects in the environment, such as the states of traffic lights(e.g., including turn signals). In addition, features of other objects(such as vehicles) that might be crossing the objects' path may also beused as input to the model.

The raw data obtained by the sensors can be processed by the perceptionsystem 224 and/or sent for further processing to the computing devices202 periodically or continuously as the data is generated by theperception system 224. For instance, the raw data from the sensors canbe quantified or arranged into a descriptive function or vector forprocessing. Computing devices 202 may use the positioning system 222 todetermine the vehicle's location and perception system 224 to detect andrespond to objects and roadway information (e.g., signage, speed limitsand/or road markings) when needed to reach the location safely, e.g.,via adjustments made by planner module 223, including adjustments inview of expected continuing lane driving behavior by another vehicle.

As illustrated in FIGS. 1A-B, certain sensors of the perception system224 may be incorporated into one or more sensor assemblies or housings.In one example, these may be integrated into front, rear or sideperimeter sensor assemblies around the vehicle. In another example,other sensors may be part of the roof-top housing (roof pod) 102. Thecomputing devices 202 may communicate with the sensor assemblies locatedon or otherwise distributed along the vehicle. Each assembly may haveone or more types of sensors such as those described above.

Returning to FIG. 2, computing devices 202 may include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as a user interfacesubsystem 234. The user interface subsystem 234 may include one or moreuser inputs 236 (e.g., a mouse, keyboard, touch screen and/ormicrophone) and one or more display devices 238 (e.g., a monitor havinga screen or any other electrical device that is operable to displayinformation). In this regard, an internal electronic display may belocated within a cabin of the vehicle (not shown) and may be used bycomputing devices 202 to provide information to passengers within thevehicle. Other output devices, such as speaker(s) 240 may also belocated within the passenger vehicle to provide information to riders,or to communicate with users or other people outside the vehicle.

The vehicle may also include a communication system 242. For instance,the communication system 242 may also include one or more wirelessconfigurations to facilitate communication with other computing devices,such as passenger computing devices within the vehicle, computingdevices external to the vehicle such as in other nearby vehicles on theroadway, and/or a remote server system. The network connections mayinclude short range communication protocols such as Bluetooth™,Bluetooth™ low energy (LE), cellular connections, as well as variousconfigurations and protocols including the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi and HTTP, and various combinations of theforegoing.

FIG. 3A illustrates a block diagram 300 with various components andsystems of a vehicle, e.g., vehicle 150 of FIGS. 1C-D. By way ofexample, the vehicle may be a truck, farm equipment or constructionequipment, configured to operate in one or more autonomous modes ofoperation. As shown in the block diagram 300, the vehicle includes acontrol system of one or more computing devices, such as computingdevices 302 containing one or more processors 304, memory 306 and othercomponents similar or equivalent to components 202, 204 and 206discussed above with regard to FIG. 2. For instance, the data mayinclude map-related information (e.g., roadgraphs), object behaviormodels, continuing lane models, etc.

The control system may constitute an electronic control unit (ECU) of atractor unit of a cargo vehicle. As with instructions 208, theinstructions 308 may be any set of instructions to be executed directly(such as machine code) or indirectly (such as scripts) by the processor.Similarly, the data 310 may be retrieved, stored or modified by one ormore processors 304 in accordance with the instructions 308.

In one example, the computing devices 302 may form an autonomous drivingcomputing system incorporated into vehicle 150. Similar to thearrangement discussed above regarding FIG. 2, the autonomous drivingcomputing system of block diagram 300 may be capable of communicatingwith various components of the vehicle in order to perform routeplanning and driving operations. For example, the computing devices 302may be in communication with various systems of the vehicle, such as adriving system including a deceleration system 312, acceleration system314, steering system 316, signaling system 318, navigation system 320and a positioning system 322, each of which may function as discussedabove regarding FIG. 2.

The computing devices 302 are also operatively coupled to a perceptionsystem 324, a power system 326 and a transmission system 330. Some orall of the wheels/tires 228 are coupled to the transmission system 230,and the computing devices 202 may be able to receive information abouttire pressure, balance, rotation rate and other factors that may impactdriving in an autonomous mode. As with computing devices 202, thecomputing devices 302 may control the direction and speed of the vehicleby controlling various components. By way of example, computing devices302 may navigate the vehicle to a destination location completelyautonomously using data from the map information and navigation system320. Computing devices 302 may employ a planner module 323, inconjunction with the positioning system 322, the perception system 324and other subsystems to detect and respond to objects when needed toreach the location safely, similar to the manner described above forFIG. 2.

Similar to perception system 224, the perception system 324 alsoincludes one or more sensors or other components such as those describedabove for detecting objects external to the vehicle, objects orconditions internal to the vehicle, and/or operation of certain vehicleequipment such as the wheels and deceleration system 312. For instance,as indicated in FIG. 3A the perception system 324 includes one or moresensor assemblies 332. Each sensor assembly 232 includes one or moresensors. In one example, the sensor assemblies 332 may be arranged assensor towers integrated into the side-view mirrors on the truck, farmequipment, construction equipment or the like. Sensor assemblies 332 mayalso be positioned at different locations on the tractor unit 152 or onthe trailer 154, as noted above with regard to FIGS. 1C-D. The computingdevices 302 may communicate with the sensor assemblies located on boththe tractor unit 152 and the trailer 154. Each assembly may have one ormore types of sensors such as those described above.

Also shown in FIG. 3A is a coupling system 334 for connectivity betweenthe tractor unit and the trailer. The coupling system 334 may includeone or more power and/or pneumatic connections (not shown), and afifth-wheel 336 at the tractor unit for connection to the kingpin at thetrailer. A communication system 338, equivalent to communication system242, is also shown as part of vehicle system 300.

Similar to FIG. 2, in this example the cargo truck or other vehicle mayalso include a user interface subsystem 339. The user interfacesubsystem 339 may be located within the cabin of the vehicle and may beused by computing devices 202 to provide information to passengerswithin the vehicle, such as a truck driver who is capable of driving thetruck in a manual driving mode.

FIG. 3B illustrates an example block diagram 340 of systems of thetrailer, such as trailer 154 of FIGS. 1C-D. As shown, the systemincludes an ECU 342 of one or more computing devices, such as computingdevices containing one or more processors 344, memory 346 and othercomponents typically present in general purpose computing devices. Thememory 346 stores information accessible by the one or more processors344, including instructions 348 and data 350 that may be executed orotherwise used by the processor(s) 344. The descriptions of theprocessors, memory, instructions and data from FIGS. 2 and 3A apply tothese elements of FIG. 3B.

The ECU 342 is configured to receive information and control signalsfrom the trailer unit. The on-board processors 344 of the ECU 342 maycommunicate with various systems of the trailer, including adeceleration system 352, signaling system 354, and a positioning system356. The ECU 342 may also be operatively coupled to a perception system358 with one or more sensors arranged in sensor assemblies 364 fordetecting objects in the trailer's environment. The ECU 342 may also beoperatively coupled with a power system 360 (for example, a batterypower supply) to provide power to local components. Some or all of thewheels/tires 362 of the trailer may be coupled to the decelerationsystem 352, and the processors 344 may be able to receive informationabout tire pressure, balance, wheel speed and other factors that mayimpact driving in an autonomous mode, and to relay that information tothe processing system of the tractor unit. The deceleration system 352,signaling system 354, positioning system 356, perception system 358,power system 360 and wheels/tires 362 may operate in a manner such asdescribed above with regard to FIGS. 2 and 3A.

The trailer also includes a set of landing gear 366, as well as acoupling system 368. The landing gear may provide a support structurefor the trailer when decoupled from the tractor unit. The couplingsystem 368, which may be a part of coupling system 334, providesconnectivity between the trailer and the tractor unit. Thus, thecoupling system 368 may include a connection section 370 (e.g., forcommunication, power and/or pneumatic links to the tractor unit). Thecoupling system also includes a kingpin 372 configured for connectivitywith the fifth-wheel of the tractor unit.

Example Scenarios

FIG. 4A illustrates a general turning scenario 400, in which a vehicle402 operating in an autonomous driving mode is traveling north on roadsegment 404. This north-south road segment intersects with east-westroad segment 406. Traveling eastbound on road segment 406 is a truck408, while another vehicle 410 approaches the intersection from asouthbound direction on road segment 404. In this example, vehicle 410may have different turning options at the intersection. It may make aleft turn as shown by solid trajectory line 412, make a right turn asshown by dashed trajectory line 414, or proceed straight as shown bydotted trajectory line 416. Whether the vehicle can turn left or rightor go straight can depend on legal lane restrictions, such as no leftturn or no turn on red. Other restrictions may be due to theconfiguration of the roadway, including the presence of lane dividers,designated turning lanes or paths, etc.

As the vehicle 402 is being maneuvered by its computing device(s), thevehicle's perception system may provide the computing devices withinformation about objects, including other road users, detected in thevehicle's surrounding environment. For instance, the perception systemmay provide the vehicle's computing device (e.g., 202 in FIG. 2 or 302in FIG. 3A) with information about truck 408 and car 410, such as theirtype, size, location, heading, velocity, lane, whether a turn signal isactive, etc.

The raw data and/or characteristics of objects received from theperception system may be used with contextual information as input to abehavior model to make a prediction about what other objects are goingto do for the predetermined period of time. For instance, informationsuch as the object's type, location, recent motion heading,acceleration, and velocity may be combined with other information, suchas whether a turn signal is activated and where the object is in theworld using the detailed map information discussed above (e.g., whichlane on a multi-lane road segment the object is traveling on). Thisinformation may be used as inputs to a behavior model. The contextualinformation may also include the status of static objects in theenvironment, such as the states of traffic lights or the presence of astop or yield sign.

The behavior model may provide a set of hypotheses, with each hypothesishaving an associated likelihood value. As an example, one or more ofthose hypotheses with the highest likelihoods (or those above aparticular threshold such that a given hypothesis is considered to havea reasonable likelihood of occurrence) may be identified as an actualfuture trajectory or behavior prediction for the object over thepredetermined period of time. For instance, the vehicle's computingdevices may use a behavior model to generate a prediction of what anobject will do during the next predetermined period of time (e.g., 2-10seconds) or a time period corresponding to how long the vehicle islikely to interact with the object (e.g., until the object is wellbehind the autonomous vehicle). Given this, the computing devices maypredict that car 410 is going to continue move towards and pass throughthe intersection according to trajectory 416, turn right according totrajectory 414, or turn left according to trajectory 412. The computingdevices may also predict whether truck 408 is going to be stopped at thetraffic light facing the eastbound lanes.

In view 420 of FIG. 4B, assume that it is predicted that the car 410will make a right turn according to trajectory 414, and that the vehicle402 will be turning left at the intersection along path 422. Therefore,according to the predicted trajectory in this example, the route 422 ofvehicle 402 and the trajectory of car 410 are expected to intersect atpoint 424. As such, a constraint may be generated for vehicle 402 toavoid the point 424, at the point in time when car 410 (or any portionof that car) is expected to overlap with point 424. This may involveslowing down or delaying entering the intersection.

FIG. 5A illustrates a continuing lane driving scenario 500 at anintersection where lanes 502L and 502R are eastbound approaching theintersection and lanes 504L, 504C and 504R are southbound approachingthe intersection. Lane 506 is eastbound heading away from theintersection, lane 508 is northbound heading away from the intersection,and lanes 510L and 51OR are westbound heading away from theintersection. In this scenario, a vehicle 512 operating in an autonomousdriving mode is in lane 502R, while a truck 514 is in lane 502L. A pairof cars 516 are in lanes 504L and 504C, while a motorcyclist 518 is inlane 504R.

As shown by the arrows along the roadway, road users in lane 502R maycontinue straight towards lane 506 or make a left along lane 508. Incontrast, lane 502L is a left turn only lane. Southbound lanes 504 havedifferent turning options. Here, lane 504L is indicated as being able togo straight only, while road users in lane 504C may go either straightor turn right (e.g., onto lane 510L). And road users in lane 504R shouldonly turn right (e.g., onto lane 510R).

If it is assumed that all road users will follow the road signs and anylegal lane restrictions, then behavior predictions for the variousobjects (e.g., truck 514, cars 516 and motorcycle 518) could bedetermined as discussed above in accordance with information obtainedfrom the vehicle's sensors as well as contextual information. However,should another road user perform a continuing lane driving behavior(e.g., continue straight from a turn-only lane, or make a turn from alane that is for continuing straight only), then such behaviorpredictions would not encompass that type of driving action. Therefore,the vehicle operating in the autonomous mode might need to quicklyrespond by braking hard and/or making an unplanned lane change. Thesereactions may have excessive impact on lateral and/or longitudinalacceleration or jerk, which can adversely affect passenger comfort.

FIG. 5B illustrates view 550, which involves a first continuing lanedriving behavior by the truck 514. Here, as noted above, lane 502L isconfigured so that road users must turn left onto lane 508. The lane502L may include indicia 552 on the road surface such as a painted arrowand/or text showing it is a left turn only lane. A traffic signal 554may include a left turn light and/or text showing it is a left turn onlylane. There may be other signage 556 along another part of theintersection indicating that the lane is not a continuing lane. Thus,the truck in this example has two options: turn left along trajectory558 onto northbound lane 508 as shown by the solid arrow, or perform acontinuing lane driving action along trajectory 560 to eastbound lane506, as shown by the dotted arrow. A roadgraph or other stored roadinformation may also indicate that lane 502L is a left turn only lanehaving an exit point that should lead to lane 508. In this example, theroadgraph would connect lane 502L with lane 508, but not with lane 506.

Should the vehicle in lane 502R predict that the truck will perform thecontinuing lane driving action, then that vehicle may modify its owndriving operation. For instance, this could include waiting at theintersection after the light turns green in order to evaluate that thetruck is indeed moving towards lane 506. Alternatively, it could includemoving forward toward lane 506 more quickly than initially planned(e.g., accelerate from a stop more quickly and/or select a driving speedthat is higher than initially planned).

FIG. 5C illustrates view 570, which involves a second continuing lanedriving behavior by the car in lane 504R. Here, as noted above, lane504R is configured so that road users must turn right onto lane 510(e.g., 510R). The lanes 504 may include indicia 572 on the road surfacesuch as a painted arrow and/or text showing it is a left turn only lane.A traffic signal 574 may include a right turn light (574 _(R)) and/ortext showing it is a left turn only lane. There may be other signagealong another part of the intersection indicating that the lane is not acontinuing lane. Thus, the car in lane 504R in this example has twooptions: turn right along trajectory 576 onto westbound lane 510 (e.g.,510R) as shown by the solid arrow, or perform a continuing lane drivingaction along trajectory 578 southbound through the intersection, asshown by the dotted arrow. A roadgraph or other stored road informationmay also indicate that lane 504R is a right turn only lane with an exitpoint that should lead to lane 510. In this example, the roadgraph wouldconnect lane 504R with lane 510 (e.g., 510R), but not with a lanesegment that passes southbound through the intersection.

Should the vehicle in lane 502R predict that the car in lane 504R willperform the continuing lane driving action southbound into theintersection, then that vehicle may modify its own driving operation.For instance, this could include waiting at the intersection after thelight turns green in order to evaluate that the car is indeed movingtowards lane 510. Alternatively, the vehicle could speed up to make sureit has cleared the intersection before the agent reaches that position.

According to aspects of the technology, machine learning approaches areused in order to obtain enhanced predictions and generate more accuratebehavior predictions for continuing lane driving situations. Thisincludes determining whether a given lane should be considered a“continuing lane”, e.g., a turn only lane having all “exits” (possibledriving locations) in the same direction. It also includes determiningwhether a continuing lane driving action should be generated for a roaduser in the continuing lane.

FIG. 6 illustrates a flow diagram 600, which illustrates one approachfor identifying possible continuing lane driving behaviors. In a firstmodule 602, information about a nearby roadgraph is fetched in block604. Nearby in this situation means within a curtain radius of theautonomous vehicle. For example, this could be within 200 meters of thevehicle. In another example, the curtain radius may be between 50-500meters. And in yet another example, the curtain radius may be within Xblocks (e.g., a 1-4 block radius). Then at block 606, for each selectedlane associated with the roadgraph, find whether there are anycorresponding continuing lane(s), which are not valid (“legal”) drivingpaths in view of the roadgraph data. In one example there may be a 1:1correspondence between the selected lane and a continuing lane. Inanother example, there may be a 1:2, 1:3 or greater correspondencebetween the selected lane and multiple continuing lanes. At block 608,the details (e.g., map information) of the continuing lane are stored inmemory. In module 610, agent data for objects in the vicinity of thevehicle (e.g., cars, motorcycles, bicycles, trucks or other road users)is obtained at block 612, such as by one or more sensors of thevehicle's perception system. The perception system and/or the onboardcomputing system may use the obtained information (e.g., position, pose,heading, speed, etc.) to identify what type of object has been detected.At block 614, the system generates continuing lane driving actions.Here, the system decides whether to generate a continuing lane drivingbehavior prediction for the agent (e.g., that the agent may elect todrive along a continuing lane). This block takes as input the agentfeatures, roadgraph features and the previously fetched candidates, andthen decides whether to generate a continuing lane driving behaviorprediction based on that. Based on this, at block 616 the systemperforms driving motion planning. In particular, this means the systemwill generate a motion plan that would encompass that driving predictionfor continuing lane driving. And at block 618, likelihood estimation isperformed. In particular, the system will assign alikelihood/probability to the prediction which indicates the probabilityof continuing lane driving happening. The likelihood estimationinformation generated at block 618 is then used by the system at block620 to generate continuing lane driving behavior predictions. This maybe used by the planner or other part of the onboard computing system tomodify autonomous driving operations of the vehicle.

FIG. 7A illustrates a flow diagram 700, which is one example fordetermining whether there is no valid (permissible) continuing laneaccording to block 606 of FIG. 6. At block 702, a decision is made as towhether a turn only lane has all the same exists in the same direction.In other words, the system checks whether this turn only lane has onlyturning lanes going out from it (exits and all turn in the samedirection). If the answer to this decision is false, then at block 704an impermissible continuing lane is not generated by the system. Becausethe system is already aware of the legal driving lanes from theroadgraph data, there is no need to run this model to determine legaldriving lanes.

At block 706, candidate continuing lanes are computed, which identifyany possible impermissible continuing lanes. At block 708, the systemruns a continuing model for all of the candidates. The continuing modelis a machine learning model. At block 710, the system (1) selects anycandidates having a score greater than a threshold, and (2) finds thecandidate having the highest score among all the candidates. If there isno candidate that satisfies these conditions, then at block 704 animpermissible continuing lane is not generated by the system. Otherwise,at block 712 the system generates a continuing lane associated with animproper (not permitted) driving operation, such as continuing straightinstead of making a turn in a turn-only lane. All (improper) continuinglanes may be stored in a cache or other memory of the system. Inputs tothe model may include any combination of source lane features, targetlane features, and related features (see example 800 of FIG. 8A). Forinstance, source lane features may include turn only lane features suchas heading, curvature, incline, camber, the presence of a traffic light,the presence of a stop sign or other signage (e.g., yield sign or noturn sign). Target lane features may include the same features. Relatedfeatures may encompass features from a combination of source and targetlanes, such as heading difference, distance between the source andtarget lanes, lateral distance, etc. In one scenario a GDBT model may beemployed; however, in other scenarios other techniques such as heatmaps, DNNs, CNNs, global nets, and/or Vector Nets could be employed. Thethreshold may be determined based on the testing framework used;however, this threshold can be tunable and hence can be set according tothe needs of the system. By way of example, the threshold may be anynumber between 0 and 1, such as 0.68, or above 0.6, or no less than0.75, etc. Scores are the output of the model. For Tree-based models,scores are the output from the leaf nodes, while for Deep net modelsthey are the output from output layers.

FIG. 7B illustrates a flow diagram 750, which is one example forgenerating impermissible continuing lane driving behavior predictionsfor one or more road users in the vehicle's nearby environment (e.g., atthe same intersection), according to block 614 of FIG. 6. At block 752,the system generates a continuing lane action for a given agent in thevehicle's vicinity (e.g., at the same intersection, within a 1-blockradius, within 200 meters, etc.). At block 754, the system fetches allnearby continuing lanes from memory, for instance which were generatedat block 712. At block 756, the system runs a continuing model for allof the candidates. The continuing model is a machine learning model thatmay be trained, for instance, using inputs as shown in example 820 ofFIG. 8B. At block 758, the system (1) selects any candidates having ascore greater than a threshold, and (2) finds the candidate having thehighest score among all the candidates. If there is no candidate thatsatisfies these conditions, then a continuing lane does not exist, andthe process terminates at block 760, where the system does not generatea continuing lane driving action (that would show an improper drivingbehavior for another road user). Otherwise, at block 762 the systemgenerates a continuing lane driving action, such as continuing straightinstead of making a turn in a turn-only lane. All (improper) continuinglane actions may be stored in a cache or other memory of the system.Here the system takes in input for the agent, the roadgraph and anycontinuing lanes derived based on the approach from FIG. 7A. In theprocess shown in FIG. 7B, the system uses features from a given roaduser/agent, roadgraph data and the continuing lane(s) under evaluation.The threshold may be determined based on the testing framework used;however, this threshold can be tunable and hence can be set according tothe needs of the system. By way of example, the threshold may be anynumber between 0 and 1, such as 0.64, or above 0.56, or no less than0.8, etc. Scores are the output of the model. For Tree-based models,scores are the output from the leaf nodes, while for Deep net modelsthey are the output from output layers.

The driving motion planner may use the continuing lane driving actionsto adjust a driving plan for the vehicle in any of the manners describedabove. The processes may be performed multiple times, for instance twoor more times a second when the vehicle operating in an autonomousdriving mode is approach an intersection or is waiting at a stop lightor a stop or yield sign. The models and information about continuinglane generation and/or continuing lane driving action generation may beshared with other vehicles, including vehicles that are part of a fleet(e.g., a fleet of ridesharing, ride hailing or delivery vehicles), aswell as with a back end system that maintains the models and associatedinformation.

Example 800 of FIG. 8A illustrates features that can be gathered andstored as training inputs used to determine whether there is no valid(permissible) continuing lane according to block 606 of FIG. 6. Here,the inputs include source lane features 802, target lane features 804,and/or related features 806. For instance, source lane features 802 mayinclude turn only lane features such as heading, curvature, incline,camber, the presence of a traffic light, the presence of a stop sign orother signage (e.g., yield sign or no turn sign). Target lane features804 may include the same types of features. Related features 806 mayencompass features based upon the relationship of source and targetlanes, such as heading difference, distance between the source andtarget lanes, lateral distance, etc.

Example 820 of FIG. 8B illustrates features that can be gathered andstored as training inputs used when determining whether to generate acontinuing lane driving action as discussed above with regard to flowdiagram 750. Here, the inputs may include lane features 822, agentfeatures 824 and/or other features 826. As noted above, the lanefeatures 822 may be one or more of the lane heading, speed limit (e.g.,posted speed limit), curvature, incline, camber, whether there is atraffic light and/or the presence of signage such as a stop or yieldsign. The agent features 824 may include one or more of recent history(e.g., what the agent has done in the last 5-30 seconds or more orless), speed, acceleration, heading, use of turn signal or hazardsignal, etc. And the other features 826 may include features based onthe relationship between the agent and the lane, such as the distancebetween the agent's current position and the lane, or a differencebetween the agent's heading and the lane heading.

The machine learning approaches discussed herein may use trained modelsin real time. However, in some situations, the approaches used todetermine whether there is no valid (permissible) continuing lane can bepreprocessed (e.g., during an earlier driving situation or received fromanother vehicle at that location) and stored in the vehicle's memory.Such stored data, which may or may not be associated with a roadgraph,can be retrieved and used when determining whether to generate acontinuing lane driving action.

In addition to using the model information for operation of the vehiclein real time, this information may also be shared with other vehicles,such as vehicles that are part of a fleet. This can be done to aid inroute planning, gathering of additional ground truth data, modelupdates, etc.

One example of data sharing is shown in FIGS. 9A and 9B. In particular,FIGS.9A and 9B are pictorial and functional diagrams, respectively, ofan example system 900 that includes a plurality of computing devices902, 904, 906, 908 and a storage system 910 connected via a network 912.System 900 also includes exemplary vehicles 914 and 916, which may beconfigured the same as or similarly to any of vehicles 100, 140 and 150.Vehicles 914 and/or vehicles 916 may be part of a fleet of vehicles.Although only a few vehicles and computing devices are depicted forsimplicity, a typical system may include significantly more.

As shown in FIG. 9B, each of computing devices 902, 904, 906 and 908 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto the ones described above with regard to FIGS. 2 and 3A-B.

The various computing devices and vehicles may communicate via one ormore networks, such as network 912. The network 912, and interveningnodes, may include various configurations and protocols including shortrange communication protocols such as Bluetooth™, Bluetooth LE™, theInternet, World Wide Web, intranets, virtual private networks, wide areanetworks, local networks, private networks using communication protocolsproprietary to one or more companies, Ethernet, WiFi and HTTP, andvarious combinations of the foregoing. Such communication may befacilitated by any device capable of transmitting data to and from othercomputing devices, such as modems and wireless interfaces.

In one example, computing device 902 may include one or more servercomputing devices having a plurality of computing devices, e.g., a loadbalanced server farm or cloud computing system, that exchangeinformation with different nodes of a network for the purpose ofreceiving, processing and transmitting the data to and from othercomputing devices. For instance, computing device 902 may include one ormore server computing devices that are capable of communicating with thecomputing devices of vehicles 914 and/or 916, as well as computingdevices 904, 906 and 908 via the network 912. For example, vehicles 914and/or 916 may be a part of one or more fleets of vehicles that can bedispatched by a server computing device to various locations. In thisregard, the computing device 902 may function as a dispatching servercomputing system which can be used to dispatch vehicles to differentlocations in order to pick up and drop off passengers and/or to pick upand deliver cargo. In addition, server computing device 902 may usenetwork 912 to transmit and present information to a user of one of theother computing devices or a passenger of a vehicle. In this regard,computing devices 904, 906 and 908 may be considered client computingdevices.

As shown in FIG. 9A each client computing device 904, 906 and 908 may bea personal computing device intended for use by a respective user 918,and have all of the components normally used in connection with apersonal computing device including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display (e.g., a monitor havinga screen, a touch-screen, a projector, a television, or other devicesuch as a smart watch display that is operable to display information),and user input devices (e.g., a mouse, keyboard, touchscreen ormicrophone). The client computing devices may also include a camera forrecording video streams, speakers, a network interface device, and allof the components used for connecting these elements to one another.

Although the client computing devices may each comprise a full-sizedpersonal computing device, they may alternatively comprise mobilecomputing devices capable of wirelessly exchanging data with a serverover a network such as the Internet. By way of example only, clientcomputing devices 906 and 908 may be mobile phones or devices such as awireless-enabled PDA, a tablet PC, a wearable computing device (e.g., asmartwatch), or a netbook that is capable of obtaining information viathe Internet or other networks.

In some examples, client computing device 904 may be a remote assistanceworkstation used by an administrator or operator to communicate withpassengers of dispatched vehicles. Although only a single remoteassistance workstation 904 is shown in FIGS. 9A-9B, any number of suchwork stations may be included in a given system. Moreover, although thework station is depicted as a desktop-type computer, the work stationmay include various types of personal computing devices such as laptops,netbooks, tablet computers, etc.

Storage system 910 can be of any type of computerized storage capable ofstoring information accessible by the server computing devices 902, suchas a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, flash drive and/ortape drive. In addition, storage system 910 may include a distributedstorage system where data is stored on a plurality of different storagedevices which may be physically located at the same or differentgeographic locations. Storage system 910 may be connected to thecomputing devices via the network 912 as shown in FIGS. 9A-B, and/or maybe directly connected to or incorporated into any of the computingdevices.

Storage system 910 may store various types of information, includingdata used as inputs to the models discussed herein, including laneinformation (e.g., source and/or target lanes), agent features and/orother features). The storage system 910 may also store autonomousvehicle control software and/or road condition models, which may be usedby vehicles, such as vehicles 914 or 916, to operate such vehicles in anautonomous driving mode. Storage system 910 may store map information(such as roadgraph information), route information, weather conditioninformation, road surface information, vehicle models for the vehicles914 and 916, weather information, etc. This information may be sharedwith the vehicles 914 and 916, for instance to help with real-time routeplanning and driving analysis by the on-board computer system(s),including when determining whether a road user may be likely to performa continuing lane driving operation.

The remote assistance workstation 904 may access the stored informationand use it to assist operation of a single vehicle or a fleet ofvehicles. By way of example, a lead vehicle may detect actions by one ormore road users at a selected intersection and send information aboutcontinuing lane driving operations to the remote assistance workstation904. In turn, the remote assistance workstation 904 may disseminate theinformation to other vehicles in the fleet, so that they may alter theirroutes or modify the models to more accurately predict continuing laneactions by other road users.

In a situation where there are passengers, the vehicle or remoteassistance may communicate directly or indirectly with the passengers'client computing device. Here, for example, information may be providedto the passengers regarding current driving operations, changes to theroute in response to the situation, etc.

FIG. 10 illustrates an example process 1000 that is a method ofoperating a vehicle in an autonomous driving mode. The method at block1002 identifies a source lane along a roadway, the source lane beingassociated with a road user other than the vehicle. At block 1004, themethod includes identifying, according to a first trained machinelearning model, a target continuing lane. The target continuing lane isan impermissible lane that the road user may move toward. At block 1006,the method includes receiving, by one or more sensors of a perceptionsystem of the vehicle, sensor data associated with objects in anexternal environment of the vehicle including the road user. At block1008, the method includes generating, according to a second trainedmachine learning model and the received sensor data, a continuing lanedriving behavior prediction for the road user along the continuing lane.And at block 1008, the method incudes controlling the vehicle in theautonomous driving mode based on the continuing lane driving behaviorprediction.

Although the technology herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent technology. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present technology as defined by the appended claims.

1. A method of operating a vehicle in an autonomous driving mode, themethod comprising: identifying, by one or more processors, a source lanealong a roadway, the source lane being associated with a road user otherthan the vehicle; identifying, by the one or more processors accordingto a first trained machine learning model, a target continuing lane, thetarget continuing lane being an impermissible lane that the road usermay move toward; receiving, by one or more sensors of a perceptionsystem of the vehicle, sensor data associated with objects in anexternal environment of the vehicle including the road user; generating,by the one or more processors according to a second trained machinelearning model and the received sensor data, a continuing lane drivingbehavior prediction for the road user along the continuing lane; andcontrolling, by the one more processors, the vehicle in the autonomousdriving mode based on the continuing lane driving behavior prediction.2. The method of claim 1, wherein identifying the source lane includesfetching map information within a curtain radius of a current locationof the vehicle.
 3. The method of claim 1, wherein identifying the targetcontinuing lane includes identifying a set of possible driving locationsexiting from the source lane.
 4. The method of claim 1, wherein, priorto controlling the vehicle, the method further includes generating amotion plan for the vehicle based at least in part on the continuinglane driving behavior prediction.
 5. The method of claim 4, furthercomprising, prior to controlling the vehicle, assigning a likelihood tothe continuing lane driving behavior prediction.
 6. The method of claim5, further comprising generating one or more continuing lane drivingbehavior predictions based on the assigned likelihood of the continuinglane driving behavior prediction.
 7. The method of claim 1, wherein thetarget continuing lane either (i) continues straight from designated aturn-only lane, or (ii) makes a turn from a lane that is designated tocontinue straight only.
 8. The method of claim 1, wherein identifyingthe target continuing lane includes: identifying any candidatecontinuing lanes; evaluating all candidate continuing lanes by the firsttrained machine learning model to identify a score for each candidatecontinuing lane; and evaluating whether the score for a given one of thecandidate continuing lanes exceeds a threshold and is higher than thescores for all other candidate continuing lanes.
 9. The method of claim8, wherein when the score for the given candidate continuing laneexceeds the threshold and is higher than the scores for all othercandidate continuing lanes, the method includes identifying the givencandidate continuing lane as the target continuing lane.
 10. The methodof claim 8, wherein when the score for the given candidate continuinglane does not exceed the threshold or is not higher than the scores forall other candidate continuing lanes, the method does not identify thegiven candidate continuing lane as the target continuing lane.
 11. Themethod of claim 1, wherein the first trained machine learning model istrained according to one or more source lane features and one or moretarget lane features.
 12. The method of claim 11, wherein the one ormore source lane features and the one or more target lane features eachinclude at least one of a lane heading, a speed limit, lane curvature,lane incline, lane camber, signage, a traffic light, or an exit point.13. The method of claim 11, wherein the first trained machine learningmodel is further trained according to one or additional featuresassociated with both a source lane and a target lane.
 14. The method ofclaim 13, wherein the one or more additional features includes at leastone of a heading difference between the source lane and the target lane,a distance between the source lane and the target lane, or a lateraldistance between the source lane and the target lane.
 15. The method ofclaim 1, wherein generating the continuing lane driving behaviorprediction includes: running the second trained machine learning modelfor each candidate continuing lane to identify all candidate continuinglane actions for the road user; evaluating all candidate continuing laneactions to identify a score for each candidate continuing lane action;and evaluating whether the score for a given one of the candidatecontinuing lane actions exceeds a threshold and is higher than thescores for all other candidate continuing lane actions.
 16. The methodof claim 15, wherein when the score for the given candidate continuinglane action exceeds the threshold and is higher than the scores for allother candidate continuing lane actions, the method includes generatingthe continuing lane driving behavior prediction according to the givencandidate continuing lane action.
 17. The method of claim 15, whereinwhen the score for the given candidate continuing lane action does notexceed the threshold or is not higher than the scores for all othercandidate continuing lane actions, the method does not generate thecontinuing lane driving behavior prediction according to the givencandidate continuing lane action.
 18. The method of claim 1, wherein thesecond trained machine learning model is trained according to one ormore lane features and one or more agent features, the one or more agentfeatures being associated with the road user.
 19. The method of claim18, wherein: the one or more lane features includes at least one of alane heading, a speed limit, lane curvature, lane incline, lane camber,signage, a traffic light, or an exit point; and the one or more agentfeatures includes at least one of an agent history, a current location,speed, acceleration, heading, use of a turn signal or use of a hazardsignal.
 20. The method of claim 19, wherein the second trained machinelearning model is further trained according to one or additionalfeatures associated with both the lane features and the agent features.21. The method of claim 20, wherein the one or more additional featuresincludes at least one of a distance between the current location of theroad user and the lane, or a difference between the heading of the roaduser and the heading of the lane.